Context:¶
The purpose is to classify a given silhouette as one of three types of vehicle, using a set of features extracted from the silhouette. The vehicle may be viewed from one of many different angles.
Attribute Information:¶
● All the features are geometric features extracted from the silhouette. ● All are numeric in nature
Objective:¶
Apply dimensionality reduction technique – PCA and train a model using principle components instead of training the model using just the raw data.
Q1. Data pre-processing – Perform all the necessary preprocessing on the data ready to be fed to an Unsupervised algorithm¶
In [102]:
#Import all the necessary modules
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
%matplotlib inline
import warnings
warnings.filterwarnings('ignore')
import seaborn as sns
from sklearn.model_selection import train_test_split
from sklearn.svm import SVC
from sklearn.decomposition import PCA
from sklearn import metrics
from sklearn.preprocessing import StandardScaler
from scipy.stats import zscore
from sklearn.metrics import accuracy_score, confusion_matrix
from sklearn.model_selection import KFold
from sklearn.model_selection import cross_val_score
In [103]:
#load the csv file vehicle.csv
df = pd.read_csv('vehicle.csv')
df.head()
Out[103]:
In [104]:
#Size of the data
df.shape
Out[104]:
In [105]:
# Data types of the fields
df.dtypes
Out[105]:
In [106]:
# Fields info
df.info()
In [107]:
df.isna().sum()
Out[107]:
Obervations:¶
- Data has total 19 fields where 18 are numeric and one is object type.
- Not all the fields has same row count. Some fields like circularity, distance_circularity,radius_ratio e.t.c has null values.
- Class is a categorical field.
In [108]:
#List all the rows having missing value in any of the single or multiple columns
missing_values_cols=df.columns[df.isnull().any()]
df[df.isnull().any(axis=1)][missing_values_cols].head()
Out[108]:
In [109]:
df[df.isnull().any(axis=1)][missing_values_cols].shape
Out[109]:
Note:¶
There are total 33 rows with missng values in one or more of 14 columns.
In [110]:
#display 5 point summary
df.describe().transpose()
Out[110]:
Univariate Distribution & Analysis using Histograms and the Boxplots¶
In [111]:
# Distribution of the data group by class variable
df['class'].value_counts()
Out[111]:
In [112]:
pd.value_counts(df["class"]).plot(kind="bar")
Out[112]:
Note: Cars are almost double in number as compared to bus and van. van is least in number
In [113]:
columns = list(df.drop(["class"],axis=1))[0:-1]
df[columns].hist(bins=10, figsize=(15,30), layout=(8,3));
In [114]:
plt.figure(figsize= (50,50))
plt.subplots_adjust(left=None, bottom=None, right=None, top=None, wspace=None, hspace=0.5)
i=1
for feature in df.drop(["class"],axis=1).columns: # Loop through all columns in the dataframe
plt.subplot(8,3,i)
b = sns.boxplot(x= df[feature])
b.set_xlabel(feature,fontsize=30)
b.tick_params(labelsize=30)
i=i+1
Observations:¶
- Seems compactness and circularity are somewhat normally distributed and no outliers found.
- distance_circularity is right skewed and no outliers.
- radius_ratio is left skewed and found some outliers.
- pr.axis_aspect_ratio most of the data in between 10 to 80. Outliers are there.
- max.length_aspect_ratio most of the data in between 3 to 15. Outliers are there.
- scatter_ratio median is left skewed. No outliers.
- elongatedness most of the data is below 45. No outliers.
- pr.axis_rectangularity is left skewed but no outliers.
- max.length_rectangularity seems somewhat normally distributed with no outliers.
- scaled_variance is left skewed with little bit of outliers.
- scaled_variance.1 is left skewed and has outliers.
- scaled_radius_of_gyration is somewhat normally distributed and no outliers.
- scaled_radius_of_gyration.1 is left skewed and most of the data between 65 and 75. Contains many outliers.
- skewness_about is left skewed and has few outliers.
- skewness_about.1 is left skewed and has few outliers.
- skewness_about.2 seems somewhat normally distributed with no outliers.
- hollows_ratio seems normally distributed without outliers.
- Many columns have long tails indicating outliers.
- Scatter ratio & Scaled_variance1 has almost perfect positive linear relationship
- Many features show high correlation indicating that we need to drop multiple features- we will use PCA for the same
In [115]:
# Let us take logaritmic transform for the below attributes which has high outliers.
# pr.axis_aspect_ratio
# max.length_aspect_ratio
# scaled_radius_of_gyration.1
df['pr.axis_aspect_ratio'] = np.log(df['pr.axis_aspect_ratio'])
df['max.length_aspect_ratio'] = np.log(df['max.length_aspect_ratio'])
df['scaled_radius_of_gyration.1'] = np.log(df['scaled_radius_of_gyration.1'])
df.head()
Out[115]:
In [116]:
# Now we will impute the missing valkues with mean of respective fields
df.fillna(df.median(),axis=0,inplace=True)
In [117]:
df.isna().sum()
Out[117]:
Null values are successfully replaced with the median values of each field.
In [118]:
##Scale the data using zscore
df_scaled =df.drop(labels='class', axis=1).apply(zscore)
df_scaled['class']=df['class']
df_scaled.head()
Out[118]:
Note:¶
- Now we can use this data in any unsupevised learing algorithms just by dropping the class variable. We have done the sclaing of the data and logerithemic transformation for the variables which has higher outliers and inputed medians in the place of null values.
Q2.Understanding the attributes - Find relationship between different attributes (Independent variables) and choose carefully which all attributes have to be a part of the analysis and why¶
Bivarient Distribution & Analysis of corelation using pair plots and the heatmap¶
In [119]:
df_scaled.corr(method='kendall')
Out[119]:
In [120]:
sns.pairplot(df,diag_kind='kde')
Out[120]:
In [121]:
corr = abs(df_scaled.corr(method='kendall')) # correlation matrix
lower_triangle = np.tril(corr, k = -1) # select only the lower triangle of the correlation matrix
mask = lower_triangle == 0 # to mask the upper triangle in the following heatmap
plt.figure(figsize = (15,8)) # setting the figure size
sns.set_style(style = 'white') # Setting it to white so that we do not see the grid lines
sns.heatmap(lower_triangle, center=0.5, cmap= 'Blues', annot= True, xticklabels = corr.index, yticklabels = corr.columns,
cbar= False, linewidths= 1, mask = mask) # Da Heatmap
plt.xticks(rotation = 50) # Aesthetic purposes
plt.yticks(rotation = 20) # Aesthetic purposes
plt.show()
Observations:¶
- From above correlation matrix we can see that there are many features which are highly correlated.
- If we observe scaled_variance.1 is highly corelated with scatter_ratio and elongatedness.
- Elongatedness is highly corelated with scatter_ration.
- pr.axis_aspect_ratio is highly corelated with scatter_ratio and elongatedness.
- scaled_radius_of_gyration.1, skewness_about, skewness_about.1 are very low corelated to any other attributes.
- There are lot of dimensions with correlation above + 0.7 and it is difficult to determine which dimensions to drop manually.
- We will use PCA to determine it.
Q3.Split the data into train and test (Suggestion: specify “random state” if you are using train_test_split from Sklearn)¶
In [122]:
#Split into Train -Test set
df_train, df_test= train_test_split(df_scaled, test_size=0.3, random_state=123)
df_train.shape, df_test.shape
Out[122]:
In [123]:
# Divide train and test set into feature and target sets
X_train=df_train.drop(labels='class', axis=1)
y_train=df_train['class']
X_test=df_test.drop(labels='class', axis=1)
y_test=df_test['class']
X_train.shape,y_train.shape, X_test.shape, y_test.shape
Out[123]:
Q4.Train a Support vector machine using the train set and get the accuracy on the test set¶
In [124]:
# Building a Support Vector Machine on train data
# svc_model = SVC(gamma = 'auto', kernel= 'poly', degree=1)
svc_model = SVC()
svc_model.fit(X_train, y_train)
y_pred = svc_model.predict(X_test)
In [125]:
# check the accuracy on the training set
print(svc_model.score(X_train, y_train))
In [126]:
# Calculate the acuracy on test dataseusing acuracy_score function of metrics
metrics.accuracy_score(y_pred,y_test)
Out[126]:
In [127]:
# Distribution of the test data group by class variable
y_test.value_counts()
Out[127]:
In [128]:
# Confusion Matrix
cm2 = confusion_matrix(pd.array(y_test), y_pred, labels=[ "car","bus", "van"])
cm2
Out[128]:
Note:¶
- Looking at the confusion matrix, model predicts cars with acuracy of 121/126 (96.03%), buses with accuracy 64/67 (95.52%) and vans with accuracy of 56/61 (91.80%)
Q5.Perform K-fold cross validation and get the cross validation score of the model¶
In [129]:
# K-fold cross validation using svc_model
num_folds = 10
seed = 7
X = df_scaled.drop(labels='class', axis=1)
Y = df_scaled['class']
kfold = KFold(n_splits=num_folds, random_state=seed)
svc_kfold = SVC()
results = cross_val_score(svc_kfold,X,Y, cv=kfold)
print(results)
print("Accuracy: %.3f%% (%.3f%%)" % (results.mean()*100.0, results.std()*100.0))
print("95 Percent Confidence Accuracy Range is Between: %.3f%% and %.3f%%" % ((results.mean()*100.0 -(1.96 * results.std()*100.0)),(results.mean()*100.0 +(1.96 * results.std()*100.0))))
Note: We can observe some growth in the accuracy percent using K-fold cross validation
Q6.Use PCA from Scikit learn, extract Principal Components that capture about 95% of the variance in the data¶
In [130]:
# Check for duplicate data
dups = df.duplicated()
print('Number of duplicate rows = %d' % (dups.sum()))
df[dups]
Out[130]:
In [131]:
# Create a covariance matrix for identifying Principal components
cov_matrix = np.cov(df_scaled.drop(labels='class', axis=1).T)
print('Covariance Matrix \n%s', cov_matrix)
print(cov_matrix.shape)
In [132]:
#Get eigen values and eigen vector
eig_vals, eig_vecs = np.linalg.eig(cov_matrix)
print('Eigen Vectors \n%s', eig_vecs)
print('\n Eigen Values \n%s', eig_vals)
In [133]:
print("Eigen Values:")
pd.DataFrame(eig_vals).transpose()
Out[133]:
In [134]:
#Find variance and cumulative variance by each eigen vector
tot = sum(eig_vals)
var_exp = [( i /tot ) * 100 for i in sorted(eig_vals, reverse=True)]
cum_var_exp = np.cumsum(var_exp) # array of size = as many PC dimensions
print("Cumulative Variance Explained", cum_var_exp)
In [135]:
plt.plot(var_exp)
Out[135]:
In [136]:
# Ploting
plt.figure(figsize=(15 , 6))
plt.bar(range(1, eig_vals.size + 1), var_exp, alpha = 0.5, align = 'center', label = 'Individual explained variance')
plt.step(range(1, eig_vals.size + 1), cum_var_exp, where='mid', label = 'Cumulative explained variance')
plt.ylabel('Explained Variance Ratio')
plt.xlabel('Principal Components')
plt.legend(loc = 'best')
plt.tight_layout()
plt.show()
Observation:¶
First 7 principal components explain more than 96.11% of the variance in the data.
In [137]:
# Use PCA command from sklearn and find Principal Components. Transform data to components formed
# NOTE - we are generating only 7 PCA dimensions (dimensionality reduction from 18 to 7)
pca = PCA(n_components=7)
data_reduced = pca.fit_transform(df_scaled.drop(labels='class', axis=1))
data_reduced.transpose()
print(data_reduced.shape)
print(data_reduced)
In [138]:
print(pca.explained_variance_)
In [139]:
print(pca.components_.shape)
pca.components_
Out[139]:
In [140]:
# Find correlation between components and features
df_comp = pd.DataFrame(pca.components_,columns=list(X_train))
df_comp
Out[140]:
In [141]:
plt.figure(figsize=(12,6))
sns.heatmap(df_comp,cmap='plasma',)
Out[141]:
Note:
- This heatmap and the color bar basically represent the correlation between the various features and the principal component itself.
- Component 3 looks more related to skewness_about.1
Q7.Repeat steps 3,4 and 5 but this time, use Principal Components instead of the original data. And the accuracy score should be on the same rows of test data that were used earlier. (hint: set the same random state)¶
In [142]:
print(df_scaled.shape)
df_scaled.head()
Out[142]:
In [143]:
##Transform original dataframe to Principal Data Components
df_scaled_pca = pd.DataFrame(pca.transform(df_scaled.drop(labels='class', axis=1)))
print(df_scaled_pca.shape)
df_scaled_pca['class']=df_scaled['class']
df_scaled_pca.head()
Out[143]:
In [144]:
#Split into Principal component data to Train -Test set
df_train_pca, df_test_pca= train_test_split(df_scaled_pca, test_size=0.3, random_state=123)
df_train_pca.shape, df_test_pca.shape
Out[144]:
In [145]:
# Divide train and test set into feature and target sets
X_train_pca=df_train_pca.drop(labels='class', axis=1)
y_train_pca=df_train_pca['class']
X_test_pca=df_test_pca.drop(labels='class', axis=1)
y_test_pca=df_test_pca['class']
X_train_pca.shape,y_train_pca.shape, X_test_pca.shape, y_test_pca.shape
Out[145]:
In [146]:
# Building a Support Vector Machine on train data
# svc_model_pca = SVC(gamma = 'auto', kernel= 'poly', degree=1)
svc_model_pca = SVC()
svc_model_pca.fit(X_train_pca, y_train_pca)
y_pred_pca = svc_model_pca.predict(X_test_pca)
In [147]:
# check the accuracy on the training set
print(svc_model_pca.score(X_train_pca, y_train_pca))
In [148]:
# Calculate the acuracy on test dataseusing acuracy_score function of metrics
metrics.accuracy_score(y_pred_pca,y_test_pca)
Out[148]:
In [149]:
# Distribution of the test data group by class variable
y_test_pca.value_counts()
Out[149]:
In [150]:
# Confusion Matrix
cm2 = confusion_matrix(pd.array(y_test_pca), y_pred_pca, labels=[ "car","bus", "van"])
cm2
Out[150]:
In [151]:
# K-fold cross validation using svc_model
num_folds = 10
seed = 7
X = df_scaled_pca.drop(labels='class', axis=1)
Y = df_scaled_pca['class']
kfold_pca = KFold(n_splits=num_folds, random_state=seed)
svc_pca_kfold = SVC()
results_pca = cross_val_score(svc_pca_kfold,X,Y, cv=kfold_pca)
print(results_pca)
print("Accuracy: %.3f%% (%.3f%%)" % (results_pca.mean()*100.0, results_pca.std()*100.0))
print("95 Percent Confidence Accuracy Range is Between: %.3f%% and %.3f%%" % ((results_pca.mean()*100.0 -(1.96 * results_pca.std()*100.0)),(results_pca.mean()*100.0 +(1.96 * results_pca.std()*100.0))))
Q8.Compare the accuracy scores and cross validation scores of Support vector machines – one trained using raw data and the other using Principal Components, and mention your findings¶
In [152]:
# Comparing the accuracy scores of train and test test data for both original and PCA data
print("Original Trained data accuracy = ", svc_model.score(X_train, y_train))
print("PCA Trained data accuracy = ",svc_model_pca.score(X_train_pca, y_train_pca))
print("-----")
print("Original test data accuracy = ",metrics.accuracy_score(y_pred,y_test))
print("PCA test data accuracy = ",metrics.accuracy_score(y_pred_pca,y_test_pca))
print("-----")
print("Original data cross validation score = ",results.mean()*100.0)
print("PCA data cross validation score = ",results_pca.mean()*100.0)
print("Original Data 95 Percent Confidence Accuracy Range is Between: %.3f%% and %.3f%%" % ((results.mean()*100.0 -(1.96 * results.std()*100.0)),(results.mean()*100.0 +(1.96 * results.std()*100.0))))
print("PCA Data 95 Percent Confidence Accuracy Range is Between: %.3f%% and %.3f%%" % ((results_pca.mean()*100.0 -(1.96 * results_pca.std()*100.0)),(results_pca.mean()*100.0 +(1.96 * results_pca.std()*100.0))))
Conclusion:¶
- From above we can see that PCA is doing a very good job.
- Accuracy with pca is approx 92.13% and with original data is approx 94.88%, but note that pca 92% accuracy is with only 7 dimensions where as original data has 18 dimensions. Means acuracy is reduced just by 2.5 percent with the Reduction of 11 dimensions.
But every thing has two sides, disadvantage of pca is we cannot do interpretation with the model. it's blackbox.
Considering the K-Fold cross validation score for original data it is 96.69% where as it is 92.67% for PCA data with same number of 10 folds. This means in production the range of accuracy with 95% confidence is between 93.442% and 99.933% with original data. For PCA data the 95% confidence accuracy range is between 87.935% and 97.407%.
In [ ]: